Realizing the Full Potential of Clinical Decision Support: Translating Usability Testing into Routine Practice in Health Care Operations

Abstract

Background Clinical Decision Support (CDS) tools have a mixed record of effectiveness, often due to inadequate alignment with clinical workflows and poor usability. While there is a consensus that usability testing methods address these issues, in practice, usability testing is generally only used for selected projects (such as funded research studies). There is a critical need for CDS operations to apply usability testing to all CDS implementations.

Objectives In this State of the Art/Best Practice paper, we share challenges with scaling usability in health care operations and alternative methods and CDS governance structures to enable usability testing as a routine practice.

Methods We coalesce our experience and results of applying guerilla in situ usability testing to over 20 projects in a 1-year period with the proposed solution.

Results We demonstrate the feasibility of adopting “guerilla in situ usability testing” in operations and their effectiveness in incorporating user feedback and improving design.

Conclusion Although some methodological rigor was relaxed to accommodate operational speed, the benefits outweighed the limitations. Broader adoption of usability testing may transform CDS implementation and improve health outcomes.

Keywords

clinical decision support - usability testing - translating research

• References

• 1 Shultz S, Hand MW. Usability: a concept analysis. . J Theory Constr Test 2015;19(02):
• 2 Department of Health and Human Services. Usability Evaluation Basics. 2013 . Accessed September 29, 2022 at: https://www.usability.gov/what-and-why/usability-evaluation.html
• 3 Nielsen Norman Group. Usability 101: Introduction to Usability. Accessed June 8, 2023 at: https://www.nngroup.com/articles/usability-101-introduction-to-usability/
• 4 Pruitt Z, Howe JL, Krevat SA, Khairat S, Ratwani RM. Development and pilot evaluation of an electronic health record usability and safety self-assessment tool. JAMIA Open 2022; 5 (03) ooac070
• 5 Park S, Marquard J, Austin R, Pieczkiewicz D, Delaney C. Usability of electronic health records from nurses' perspectives: a systematic review. Paper presented at: 2022 IEEE 10th International Conference on Healthcare Informatics (ICHI). 2022: 511-512
• 6 Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA 2018; 319 (12) 1276-1278
• 7 Yen PY, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012; 19 (03) 413-422
• 8 Melnick ER, Dyrbye LN, Sinsky CA. et al. The association between perceived electronic health record usability and professional burnout among US physicians. Mayo Clin Proc 2020; 95 (03) 476-487
• 9 Khairat S, Burke G, Archambault H, Schwartz T, Larson J, Ratwani RM. Perceived burden of EHRs on physicians at different stages of their career. Appl Clin Inform 2018; 9 (02) 336-347
• 10 Gomes KM, Ratwani RM. Evaluating improvements and shortcomings in clinician satisfaction with electronic health record usability. JAMA Netw Open 2019; 2 (12) e1916651
• 11 Koppel R, Metlay JP, Cohen A. et al. Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005; 293 (10) 1197-1203
• 12 Ratwani R, Fairbanks T, Savage E. et al. Mind the Gap. A systematic review to identify usability and safety challenges and practices during electronic health record implementation. Appl Clin Inform 2016; 7 (04) 1069-1087
• 13 Dixit RA, Boxley CL, Samuel S, Mohan V, Ratwani RM, Gold JA. Electronic health record use issues and diagnostic error: a scoping review and framework. J Patient Saf 2023; 19 (01) e25-e30
• 14 Kariotis TC, Prictor M, Chang S, Gray K. Impact of electronic health records on information practices in mental health contexts: scoping review. J Med Internet Res 2022; 24 (05) e30405
• 15 Li RC, Wang JK, Sharp C, Chen JH. When order sets do not align with clinician workflow: assessing practice patterns in the electronic health record. BMJ Qual Saf 2019; 28 (12) 987-996
• 16 Reese TJ, Liu S, Steitz B. et al. Conceptualizing clinical decision support as complex interventions: a meta-analysis of comparative effectiveness trials. J Am Med Inform Assoc 2022; 29 (10) 1744-1756
• 17 Orenstein EW, Weitkamp AO, Rosenau P, Mallozzi C, Tobias MC. Towards a maturity model for clinical decision support operations: an interactive panel. Paper presented at: American Medical Informatics Association Clinical Informatics Conference. Atlanta, GA; 2019
• 18 Kushniruk A, Borycki E, Kuo MH, Kuwata S. Integrating technology-centric and user-centric system testing methods: ensuring healthcare system usability and safety. In: Information Technology in Health Care: Socio-Technical Approaches 2010. IOS Press; 2010: 181-186 . Accessed November 8, 2023 at: https://ebooks.iospress.nl/doi/10.3233/978-1-60750-569-3-181
• 19 Desai AV, Michael CL, Kuperman GJ. et al. A novel patient values tab for the electronic health record: a user-centered design approach. J Med Internet Res 2021; 23 (02) e21615
• 20 Blanes-Selva V, Asensio-Cuesta S, Doñate-Martínez A, Pereira Mesquita F, García-Gómez JM. User-centred design of a clinical decision support system for palliative care: Insights from healthcare professionals. Digit Health 2023; 9: 20 552076221150735
• 21 Rudin RS, Perez S, Rodriguez JA. et al. User-centered design of a scalable, electronic health record-integrated remote symptom monitoring intervention for patients with asthma and providers in primary care. J Am Med Inform Assoc 2021; 28 (11) 2433-2444
• 22 Horsky J, Ramelson HZ. Development of a cognitive framework of patient record summary review in the formative phase of user-centered design. J Biomed Inform 2016; 64: 147-157
• 23 Chokshi SK, Belli HM, Troxel AB. et al. Designing for implementation: user-centered development and pilot testing of a behavioral economic-inspired electronic health record clinical decision support module. Pilot Feasibility Stud 2019; 5 (01) 28
• 24 Thursky KA, Mahemoff M. User-centered design techniques for a computerised antibiotic decision support system in an intensive care unit. Int J Med Inform 2007; 76 (10) 760-768
• 25 Nguyen KA, Patel H, Haggstrom DA, Zillich AJ, Imperiale TF, Russ AL. Utilizing a user-centered approach to develop and assess pharmacogenomic clinical decision support for thiopurine methyltransferase. BMC Med Inform Decis Mak 2019; 19 (01) 194
• 26 Miller SD, Murphy Z, Gray JH. et al. Human-centered design of a clinical decision support for anemia screening in children with inflammatory bowel disease. Appl Clin Inform 2023; 14 (02) 345-353
• 27 Molloy MJ, Zackoff M, Gifford A. et al. Usability testing of situation awareness clinical decision support in the intensive care unit. Appl Clin Inform 2024; 15 (02) 327-334
• 28 Shear K, Rice H, Garabedian PM. et al. Usability testing of an interoperable computerized clinical decision support tool for fall risk management in primary care. Appl Clin Inform 2023; 14 (02) 212-226
• 29 McGonagle EA, Karavite DJ, Grundmeier RW. et al. Evaluation of an antimicrobial stewardship decision support for pediatric infections. Appl Clin Inform 2023; 14 (01) 108-118
• 30 McNab D, Freestone J, Black C, Carson-Stevens A, Bowie P. Participatory design of an improvement intervention for the primary care management of possible sepsis using the Functional Resonance Analysis Method. BMC Med 2018; 16 (01) 174
• 31 Beerlage-de Jong N, Wentzel J, Hendrix R, van Gemert-Pijnen L. The value of participatory development to support antimicrobial stewardship with a clinical decision support system. Am J Infect Control 2017; 45 (04) 365-371
• 32 van Leeuwen D, Mittelman M, Fabian L, Lomotan EA. Nothing for me or about me, without me: codesign of clinical decision support. Appl Clin Inform 2022; 13 (03) 641-646
• 33 Kushniruk A, Nohr C. Participatory design, user involvement and health IT evaluation. In: Evidence-Based Health Informatics. IOS Press; 2016. :139–151. Accessed July 30, 2024 at: https://ebooks.iospress.nl/doi/10.3233/978-1-61499-635-4-139
• 34 Perry SJ, Catchpole K, Rivera AJ, Henrickson Parker S, Gosbee J. 'Strangers in a strange land': Understanding professional challenges for human factors/ergonomics and healthcare. Appl Ergon 2021; 94: 103040
• 35 Nielsen J, Levy J. Measuring usability: preference vs. performance. Commun ACM 1994; 37 (04) 66-75
• 36 Staggers N, Rodney M. Promoting Usability in Organizations with a New Health Usability Model: Implications for Nursing Informatics. NI 2012: 11th International Congress on Nursing Informatics; June 23–27, 2012; Montreal, Canada. 2012: 396 . Accessed June 8, 2023 at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3799150/
• 37 Shah T, Kitts AB, Gold JA. et al. Electronic health record optimization and clinician well-being: a potential roadmap toward action. NAM Perspect 2020; 2020
• 38 Mann DM, Chokshi SK, Kushniruk A. Bridging the gap between academic research and pragmatic needs in usability: a hybrid approach to usability evaluation of health care information systems. JMIR Hum Factors 2018; 5 (04) e10721
• 39 Kushniruk AW, Borycki EM, Kannry J. Commercial versus in-situ usability testing of healthcare information systems: towards “public” usability testing in healthcare organizations. Stud Health Technol Inform 2013; 183: 157-161
• 40 Nielsen Norman Group. Discount Usability: 20 Years. Accessed May 14, 2024 at: https://www.nngroup.com/articles/discount-usability-20-years/
• 41 Motulsky SL. Is member checking the gold standard of quality in qualitative research?. Qual Psychol 2021; 8 (03) 389-406
• 42 Holden RJ, Carayon P. SEIPS 101 and seven simple SEIPS tools. BMJ Qual Saf 2021; 30 (11) 901-910
• 43 McDonald S, Edwards HM, Zhao T. Exploring Think-Alouds in Usability Testing: an international survey. IEEE Trans Prof Commun 2012; 55 (01) 2-19
• 44 EDawson T, Beus J, WOrenstein E, Umontuen U, McNeill D, Kandaswamy S. Reducing therapeutic duplication in inpatient medication orders. Appl Clin Inform 2023; 14 (03) 538-543
• 45 Orenstein EW, Boudreaux J, Rollins M. et al. Formative usability testing reduces severe blood product ordering errors. Appl Clin Inform 2019; 10 (05) 981-990
• 46 Orenstein EW, Rollins M, Jones J. et al. Influence of user-centered clinical decision support on pediatric blood product ordering errors. Blood Transfus Trasfus Sangue 2023; 21: 3-12
• 47 Thompson SA, Williams H, Rzewnicki D. et al. Avoiding Unintended Consequences of Pediatric Blood Order Set Updates through In Situ Usability Testing. Appl Clin Inform. Accessed July 23, 2024 at: https://www.thieme-connect.com/products/ejournals/abstract/10.1055/a-2351-9642
• 48 Kandaswamy S, Jones J, Orenstein EW. User Center Design of Blood Orders in the EHR to reduce risk of over-transfusion in a pediatric healthcare system. Paper presented at : AABB Annual Meeting. Virtual Meeting: Wiley Online Library; 2020
• 49 Orenstein EW, Rasooly IR, Mai MV. et al. Influence of simulation on electronic health record use patterns among pediatric residents. J Am Med Inform Assoc 2018; 25 (11) 1501-1506
• 50 Orenstein EW, ElSayed-Ali O, Kandaswamy S. et al. Evaluation of a clinical decision support strategy to increase seasonal influenza vaccination among hospitalized children before inpatient discharge. JAMA Netw Open 2021; 4 (07) e2117809-e2117809
• 51 Mrosak J, Kandaswamy S, Stokes C. et al. The influence of integrating clinical practice guideline order bundles into a general admission order set on guideline adoption. JAMIA Open 2021; 4 (04) ooab087
• 52 Mrosak J, Kandaswamy S, Stokes C, Roth D, Orenstein E. Formative usability testing of an admission order set with modular disease-specific order groups to improve guideline order set usage. Paper presented at: Pediatric Hospital Medicine Conference. 2020. ; Florida
• 53 Kandaswamy S, Gill A, Wood S. et al. User-centered design of central venous access device documentation. JAMIA Open 2022; 5 (01) ooac011
• 54 Kandaswamy S, Jain S, Dwight C. et al. Improving outcomes for pediatric patients with metabolic conditions in the ED using user centered design for order sets. Paper presented at: AMIA 2021 Annual Symposium. November 30, 2021 ; San Diego.
• 55 Bennett B, Kurzen E, Orenstein E, Kandaswamy S, Shin H. Clinical decision support to reduce nephrotoxic medication-associated acute kidney injury in non-critically ill hospitalized children. Paper presented at: AMIA Annual Meeting 2022. November 5, 2022 ; Washington, D.C.
• 56 Rowland AF, Nguyen TH, Cunha PP. et al. Implementing a clinical decision support tool to increase early peanut introduction guidance. J Allergy Clin Immunol 2024; S0091 -6749(24)00712-7
• 57 Thompson SA, Kandaswamy S, Orenstein E. CIC 2023: A discount approach to reducing nursing alert burden. Appl Clin Inform 2024; 15: 727-732
• 58 Siegel BI, Johnson M, Dawson TE. et al. Reducing prescribing errors in hospitalized children on the ketogenic diet. Pediatr Neurol 2021; 115: 42-47
• 59 Mrosak J, Kandaswamy S, Stokes C. et al. The effect of implementation of guideline order bundles into a general admission order set on clinical practice guideline adoption: quasi-experimental study. JMIR Med Inform 2023; 11: e42736
• 60 Kandaswamy S, Masterson E, Blanco R. et al. Barriers to seasonal influenza vaccine uptake in a pediatric inpatient healthcare setting after implementation of clinical decision support. Stud Health Technol Inform 2022; 290: 452-456
• 61 Siegel B, Shahnawaz M, Elkins K, Kheder A, Orenstein E. Validation of a semi-automated method for tracking quality metrics in benzodiazepine-resistant status epilepticus (S7.007). . Neurology 2022;98(18 Supplement). Accessed June 13, 2023 at: https://n.neurology.org/content/98/18_Supplement/706
• 62 Thompson S, Mckay L, Ray E, Kandaswamy S, Orenstein EW. Improve or remove: challenges in refining ineffective alerts – the case of elopement prevention. Paper presented at: AMIA 2023 Clinical Informatics Conference. May 23, 2023 ; Chicago, IL.
• 63 Nguyen TH, Cunha PP, Rowland AF, Orenstein E, Lee T, Kandaswamy S. User-centered design and evaluation of clinical decision support to improve early peanut introduction: formative study. JMIR Form Res 2023; 7 (01) e47574
• 64 Cunha P, Nguyen T, Pham T, Kandaswamy S, Lee T. Implementing early peanut introduction recommendations by pediatric residents through a clinical decision support system. J Allergy Clin Immunol 2023; 151 (02) AB44 . Accessed March 10, 2023 at: https://www.jacionline.org/article/S0091-6749(22)01792-4/fulltext
• 65 Dawson TE, Beus JM, Orenstein E, Kurzen E, Kandaswamy S. Decision support to improve safety of discharge prescriptions of subcutaneous medications. Paper presented at: AMIA 2023 Clinical Informatics Conference. May 23, 2023 ; Chicago, IL.
• 66 Kandaswamy S, Thompson S, Orenstein E. Accurate dosing weight: when the 10% really matters. Stud Health Technol Inform 2024; 310: 354-358
• 67 Kandaswamy S, Dawson TE, Shin H, Orenstein EW. Non-interruptive alerts for reduced renal function: does anyone see them?. Paper presented at: AMIA 2023 Clinical Informatics Conference. May 23, 2023 ; Chicago, IL.
• 68 Thompson S, Orenstein EW, Kandaswamy S. Accurate dosing weight: when the 10% really matters. Paper presented at: AMIA 2023 Clinical Informatics Conference. May 23, 2023 ; Chicago, IL.
• 69 Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006; 3: 77-101 Accessed May 14, 2024 at: https://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa
• 70 Stead WW, Haynes RB, Fuller S. et al. Designing medical informatics research and library–resource projects to increase what is learned. J Am Med Inform Assoc 1994; 1 (01) 28-33
• 71 Kushniruk A. Evaluation in the design of health information systems: application of approaches emerging from usability engineering. Comput Biol Med 2002; 32 (03) 141-149
• 72 Kaufman D, Roberts WD, Merrill J, Lai TY, Bakken S. Applying an evaluation framework for health information system design, development, and implementation. Nurs Res 2006; 55 (2 Suppl): S37-S42
• 73 Ammenwerth E, Gräber S, Herrmann G, Bürkle T, König J. Evaluation of health information systems-problems and challenges. Int J Med Inform 2003; 71 (2–3): 125-135